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The  Role  of  Simulation  in  Test  and  Evaluation 

Major  Cornelius  L.  Allen,  Jr. 

U.S.  Army  Operational  Test  Command,  West  Fort  Hood,  Texas 

Simulation  has  an  essential  role  in  test  and  evaluation.  It  provides  efficiencies  in  U.S.  Army 
acquisition  strategies  of  mission-based  test  and  evaluation  and  integrated  testing  and  training, 
as  evident  with  the  Boeing  Engineering  Development  Simulator  during  the  Apache  Block  III 
force  development  test  and  experimentation  testing.  Simulators  provide  a  representative 
environment  where  testers  can  safely  test  a  product’s  or  systems  mission  effectiveness.  Ideal  for 
integrated  testing,  simulators  combine  developmental  and  operational  testing  as  they  enable  safe 
simultaneous  testing  of  multiple  elements.  Additional  benefits  of  cost  and  risk  reduction  were 
realized  as  well  as  aviation  doctrine  development. 

Key  words:  Acquisition  strategies;  aviation  combined  arms  tactical  trainer  (AVCATT); 
Boeing  Engineering  Development  Simulator;  hardware  development;  realistic  environ¬ 
ment;  risk  reduction;  safety;  software  code  development;  test  and  training. 


As  a  new  U.S.  Army  aviator,  I  immedi¬ 
ately  learned  the  importance  of  simula¬ 
tors  in  both  initial  and  proficiency  flight 
training.  Their  significance  became 
readily  apparent  recently  when,  as  an 
acquisitions  test  and  evaluation  officer,  I  conducted  two 
simulation-centric  tests.  The  first  test  focused  on  the 
fidelity  of  the  aviation  combined  arms  tactical  trainer 
(AVCATT)  training  simulator,  and  the  second  dealt 
with  the  utilization  of  the  simulator  to  validate  tactical 
employment  methods  of  new  functionalities  of  the  AH- 
64D  Apache  Block  III  (AB3).  This  article  identifies  the 
test  and  evaluation  (T&E)  and  training  benefits  of 
simulators  using  the  Boeing  Engineering  Development 
Simulator  (EDS)  as  a  case  study. 

The  AVCATT  customer  test’s  purpose  was  to 
measure  the  fidelity  of  a  software  lot  upgrade.  The 
test  focused  on  the  system’s  ability  to  replicate  the 
Army’s  current  fleet.  AVCATT  is  a  high-fidelity,  full- 
mission  simulator  with  reconfigurable  cockpits  that 
represent  any  of  the  five  Army  attack,  reconnaissance, 
and  utility  helicopters;  specifically,  for  this  test  it 
represents  the  AH-64D  Apache  Longbow  and  OH- 
58D  Kiowa.  For  the  user,  the  AVCATT  is  an  air 
mission  commander’s  tool  to  exercise  command  and 
control  operations  to  conduct  company/ troop-level 
battle  drills.  Developers  utilize  customer  tests  to  assess 
a  product-  or  system-readiness  state  and  its  effective¬ 
ness  by  placing  it  in  the  hands  of  the  user  (soldiers). 

Simulation’s  role  in  T&E  was  most  significant  in  the 
AB3  Force  Development  Test  and  Experimentation 


(FDT&E)  in  October  2009  at  the  Boeing  facility,  Mesa, 
Arizona,  utilizing  the  EDS.  The  AB3  is  the  upgraded 
version  of  the  currently  fielded  AH-64D  Apache 
Longbow.  The  primary  purpose  was  to  validate  aircrew 
and  teaming,  reconnaissance  and  attack  techniques, 
tactics  and  procedures  (TTPs)  that  incorporate  the 
Unmanned  Aircraft  System  (UAS)  level  of  interopera¬ 
bility  (LOI)  II— IV,  instrument  flight  rule,  and  other 
functionalities.  The  objective  was  achieved  along  with  a 
windfall  of  other  benefits  to  AB3  program,  reconnais¬ 
sance/attack  aviation,  and  Army  aviation  doctrine.  The 
most  pertinent  was  refinement  of  the  TTPs  in  manned- 
unmanned  teaming  operations. 

UAS  levels  of  interoperability  (Dunbar 
2010) 

•  LOI  I:  indirect  receipt  of  secondary  imagery  or 
data, 

•  LOI  II:  receipt  of  imagery  or  data  directly  from 
the  UAS, 

•  LOI  III:  LOI  II  plus  control  of  the  UAS  sensor, 

•  LOI  IV:  LOI  III  plus  navigational  control  of  the 
UAS. 

Mission-based  test  and  evaluation 
(MBT&E):  Focusing  the  data  collection 
effort  on  mission  effectiveness 

For  the  unfamiliar,  MBT&E  is  the  Department  of 
the  Army/Office  of  Secretary  of  Defense  standard  for 
testing  to  ensure  a  system  of  systems  or  family  of 
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Simulation  in  T&E 


Figure  1.  EDS  cockpit  (photo  by  Mike  Goettings). 


systems  meets  the  user’s  needs  (mission  task  capabil¬ 
ities)  as  opposed  to  just  “checking  the  blocks”  on  a 
requirement  list.  When  done  correctly,  the  end  result  is 
an  increase  in  the  warfighter’s  ability  to  accomplish  his/ 
her  mission.  The  testers  and  evaluators  must  ask 
themselves,  “Is  the  desired  effect  achieved”?  To  answer 
this  question  the  test  article  must  be  examined  in  a 
representative  operational  environment;  this  is  where 
the  EDS  enters  into  the  equation.  The  consensus  of 
the  test  team  and  test  players  supported  that  the  EDS 
not  only  met  but  exceeded  test  expectations. 

The  EDS  is  a  high-fidelity  dome  simulator  with  two 
independent  but  networked  cockpits.  From  the  pilot’s 
perspective,  the  cockpits  are  to-scale  functional  dupli¬ 
cates  of  the  actual  article.  They  look,  feel,  sound,  and 
interact  with  the  crew  member  just  as  the  aircraft  does 
to  provide  representative  human-machine  interface 
data.  Limited-motion  cueing  and  vibration  are  pro¬ 
duced  under  the  crew  member  seats.  Typical  aircraft 
noise  is  provided  from  the  appropriate  direction  and 
intensity  level  by  a  surround  sound  audio  system. 
Recording  capability  of  the  crew  station  multipurpose 
displays  is  available  as  well  as  from  over-the-shoulder 
cameras.  Visually,  the  cockpit  is  encompassed  by  large, 
high-resolution  screens  that  represent  the  Apache 
tandem  cockpit  field  of  view.  Inclement  weather  or 
environmental  cues  such  as  shadow,  sun,  and  moon 
position  are  present.  Additionally,  battlefield  effects 
such  as  smoke,  explosion,  and  fire  are  used  to  enhance 
realism.  The  EDS  is  supported  by  an  auditorium  that 
can  display  the  crew  member’s  multipurpose  displays 
and  a  tactical  map  ( Figures  1  and  2). 

The  instrument  of  execution,  the  EDS  was  more 
than  capable  of  providing  the  virtual  mission  vignettes 
that  replicate  past,  present,  and  potential  threat 
scenarios  as  identified  in  the  operational  mission 
statement/mission  profile  document.  Vignette  devel¬ 
opment  incorporated  input  from  various  Department 
of  Defense  agencies  to  ensure  all  known  and  forecasted 


Figure  2.  Auditorium  (from  Ferguson  2009). 


variables  were  addressed.  First,  the  mission  vignettes 
were  crafted  to  test  AB3’s  enhanced  capabilities  in  the 
traditional  fight.  The  traditional  fight  is  defined  as  a 
contiguous  force-on-force  scenario  against  a  developed 
nation’s  army.  The  second  objective  was  to  determine 
the  effectiveness  of  the  aircraft  capabilities  in  today’s 
contemporary  operating  environment.  The  contempo¬ 
rary  operating  environment  was  representative  of  the 
insurgent  fight  in  Afghanistan  and  Iraq  theaters  of 
operations.  A  hybrid  fight  was  incorporated  in  between 
for  a  transitional  phase,  characterized  as  a  simultaneous 
fight  against  both  a  defeated  standing  force  and  a 
constituting  insurgent  force. 

The  most  apparent  advantage  of  the  simulator  is  that 
it  provides  a  substantial  amount  of  data  compared  to  an 
aircraft-based  event  and  a  realistic  combat  theater 
battle  rhythm.  A  mission  set  was  assigned  to  each  of 
three  aircrews  throughout  an  eight-day  record  test.  A 
total  of  23  operational  missions  were  conducted,  with 
the  24th  mission  reserved  as  a  test/verification  (prove- 
out)  mission  to  resolve  any  tactical  steering  committee 
changes.  Mission  preparation,  including  operation 
orders,  intelligence  briefs,  rules  of  engagement,  air 
mission  briefs,  rehearsals,  and  mission  debriefs,  were 
conducted  in  accordance  with  a  daily  combat  battle 
rhythm.  A  total  of  75.6  simulator  flight  hours  were 
flown  during  the  24  missions.  Compared  to  using 
actual  aircraft,  the  EDS  obviously  facilitated  a  more 
robust  schedule  ( Figure  3).  The  EDS  overlapping 
mission  cycles  were  more  representative  than  a  typical 
single-mission-per-day  aircraft  event.  The  operational 
tempo  generated  an  optimum  data  collecting  effort 
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Figure  3.  FDT&E  daily  schedule. 


while  maximizing  resources  in  a  significantly  safer 
environment. 

FDT&E  and  limited  user  test 
(LUT)  comparison 

•  Test  team  personnel:  15  versus  103, 

•  FDT&E  test  cost:  5  percent  of  LUT  cost, 

•  risk  assessment  level:  low  versus  high, 

•  record  test  days:  eight  versus  nine, 

•  record  data  collection  events  (missions):  24  versus 
nine. 

High  fidelity  was  achieved  by  seamless  integration  of 
individual  player  entities  into  a  multifacet  battlefield  by 
the  operations  officer  from  the  simulator  control  room. 
The  key  to  a  dynamic  battle  simulation  was  having  the 
right  personnel  (role  players)  in  the  room  (see  control 
room  personnel  list  below).  Through  the  operations 
officer,  control  room  personnel  had  vetted  interactions 
with  the  battle  simulation  and  aircrew.  To  establish 
tactical  realism,  the  operations  officer  was  centrally 
located  during  the  vignettes  to  orchestrate  the  event 
while  reducing  game-ism  as  much  as  possible.  After 
sequencing  of  role  player  actions,  filtering  simulation 
distractions  and  nuances  from  the  aircrew  was  the 
operations  officer’s  next  most  important  task.  Con¬ 
versely,  other  distractions  such  as  sketchy  radio 
communications  and  incorrect  intelligence  information 
were  incorporated  for  imperfect  conditions — fog  and 
friction  of  war.  A  threat  analyst  ensured  simulated 
enemy  tactics  and  systems  were  employed  in  accor¬ 
dance  with  their  doctrine  and  recent  Afghanistan  and 
Iraq  battlefield  reports.  Ranges  for  probability  of  hit 


and  kill  were  in  line  with  demonstrated  enemy 
capability.  A  training  and  doctrine  capabilities  manager 
recon/attack  subject  matter  expert  served  as  an  honest 
broker  to  adjudicate  weapon  engagements.  Video 
record  capability  of  the  EDS  was  instrumental  in 
mission  after-action  review,  continuous  TTP  refine¬ 
ment,  and  test  team  after-action  review  to  ensure  the 
test  team  was  synchronized. 

EDS  control  room  personnel 

•  Operations  officer; 

•  test  officer; 

•  subject  matter  expert; 

•  UAS  ground  control  station  operators; 

•  friendly  element  controller: 

o  infantry,  armor,  artillery,  and  joint  services; 

•  AH-64D  Apache  wingman  (laptop  networked); 

•  aviation  tactical  operation  control; 

•  Enemy  force  commander: 

o  infantry,  armor,  artillery,  and  insurgents; 

•  Threat  analyst; 

•  EDS  operator; 

•  EDS  support  personnel: 

o  simulator  operations,  environment  conditions, 
and  noncombatants. 

Integrated  testing:  Fusing  developmental 
and  operational  testing  in  one  event 

Integrating  developmental  and  operational  testing 
captures  test  efficiencies  by  making  use  of  all  data 
available  throughout  testing  as  opposed  to  two  distinct 
test  phases.  Integrated  testing  is  commonly  achieved  by 
increasing  operational  realism  in  developmental  test- 
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Figure  4.  Unmanned  Little  Bird  (photo  by  Mike  Goettings). 


ing,  therefore,  improving  the  likelihood  of  success  in 
subsequent  operational  tests  and  ultimately  in  combat. 
Direct  user  feedback  early  in  developmental  testing 
greatly  reduces  the  cost  of  modification.  Simulators  are 
ideal  vehicles  for  integrated  testing,  as  they  enable 
parallel  testing  of  multiple  elements  without  adding 
risk.  Simultaneous  testing  of  a  developing  system  and 
its  application  not  only  reduces  resource  consumption 
but  also  optimizes  its  employment  once  fielded.  This 
was  an  operational  event  with  a  substantial  amount  of 
developmental  testing  incorporated. 

The  ability  to  temporarily  bypass  an  immature  key 
component  during  developmental  testing  is  a  time  and 
cost  benefit  that  simulation  provides  to  complex 
programs.  In  the  AB3  program,  the  EDS  allowed 
the  testing  of  capability  that  has  yet  to  reach  fruition. 
With  the  Extended  Range/Multi-Purpose  UAS  still  in 
development,  it  was  advantageous  to  conduct  the 
FDT&E  in  virtual  simulation.  The  LUT  utilized  an 
experimental  aircraft,  a  modified  Boeing  AH-6  known 
as  the  Unmanned  Little  Bird  (Figure  4).  The  EDS 
allowed  the  program  to  proceed  on  schedule  while  a 
surrogate  for  the  UAS,  which  was  required  for  the 
LUT,  was  under  development. 

Operationally,  the  effects  of  the  AB3  functions  on 
workload  and  situational  awareness  were  analyzed  in 
addition  to  airspace  management  and  engagement 
geometry.  Multiple  functions  of  air  weapon  system 
employment  were  examined  simultaneously  in  the 
safety  of  a  virtual  environment.  The  currently  fielded 
Apache  is  a  complex  weapon  system  with  a  high 
workload.  Introduction  of  the  additional  task  of 
controlling  a  UAS  in  teaming  operations  was  a  critical 
concern  of  the  aviation  community.  The  EDS  provided 
a  means  of  direct  comparison  of  legacy  Apache  to  AB3 
crew  member  workload,  situational  awareness,  and 
mission  effectiveness.  The  Army  Research  Laboratory- 
Human  Research  &  Engineering  Directorate  repre¬ 


sentative  was  able  to  isolate  when  or  if  a  crew  member 
became  task  saturated  or  lost  situational  awareness. 
The  tactical  steering  committee  used  these  observa¬ 
tions  to  change  intracockpit  delegation,  task  division, 
and  realignment  of  crew  duties. 

The  greatest  risk  to  dissimilar  aircraft  operating  in 
teaming  is  other  aircraft.  Therefore,  AB3s  airspace 
management  was  a  critical  element  of  TTP  develop¬ 
ment,  reducing  the  risk  of  placing  mixed  aircraft  teams 
in  close  proximity  of  each  other.  At  this  point,  the 
TTPs  moved  from  concept  to  a  tangible  employment 
method.  Tactically  in  cooperative  engagements,  the 
geometry  of  the  (designating)  UAS  in  relation  to  the 
(firing)  AB3  was  honed  to  definitive  envelope  to  be 
field  tested.  Equally  important,  the  most  efficient 
(quickest)  method  to  achieve  the  correct  geometry  was 
refined.  Appropriate  engagement  geometry  requires 
both  aircraft  to  have  the  correct  combination  of  several 
factors,  such  as  slant  range;  engagement  fan  (azimuth); 
air  speed;  timing;  and  loitering  patterns  to  acquire, 
designate,  and  process  targets. 

The  AB3  FDT&E  leveraged  developmental  testing 
of  both  software  and  hardware.  The  EDS  serves  as  a 
continuous  test  bed  for  the  software  code  development, 
since  it  uses  exact  production  representative  compo¬ 
nents  and  processors  (black  boxes)  as  the  AB3. 
Engineers  routinely  use  the  EDS  as  a  code  prove-out 
vehicle  to  answer  “what-ifs”  and  bring  anomalies  and 
glitches  to  light.  In  hardware  development,  the  pilots 
were  fitted  with  modified  HGU-56/P  as  opposed  to 
the  currently  fielded  Integrated  Helmet  and  Display 
Sight  System.  The  modified  HGU-56/P  helmet  offers 
the  same  mounted  monocular  helmet  display  unit  with 
flight  and  targeting  symbology  overlays.  A  new  helmet 
system,  range  of  motion,  sensor  location,  cord 
restrictions,  field  of  view,  and  fitting  are  critical  issues 
of  pilot-vehicle  interaction  that  were  examined  in 
depth.  Other  than  the  aircraft,  the  EDS  is  the  only 
environment  where  the  pilots  perform  normal  opera¬ 
tional  head  movements.  Two  pilots  stated  the  HGU- 
5 6/P  helmets  compared  to  the  current  Integrated 
Helmet  and  Display  Sight  System  helmet  as  the  best 
fit  in  20  years  of  aviation  flying  service. 

Training:  Test  benefits  and  more 

Beyond  the  FDT&E  test,  the  EDS  contribution  to 
the  Apache  Block  III  limited-user  test  was  invaluable. 
LUT  pilots  underwent  a  two-week  training  phase 
consisting  of  academics,  EDS  sessions,  and  training 
flights  in  the  developmental  AB3  aircraft.  The  LUT 
aircrews  reinforced  their  academics  with  60  cumulative 
flight  hours  of  EDS  practical  (operational)  application, 
all  of  whom  stated  that  using  the  EDS  as  a  training  aid 
was  advantageous.  Both  FDT&E  and  LUT  test  team 
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and  players  recommended  the  use  of  the  EDS  as  a 
training  aid  in  future  testing  as  well  as  unit  fielding. 
Mission  segments  from  the  FDT&E  missions  were 
recorded  and  incorporated  into  the  LUT  refresher 
training.  At  the  conclusion  of  the  FDT&E,  a  training 
program  was  finalized  for  LUT  and  first-unit- 
equipped  pilots.  Also,  training  of  the  test  players 
identified  gaps  to  be  incorporated  into  the  Apache 
Longbow  Crew  Trainer  and  the  new  equipment 
training  package.  Training  was  not  limited  to  the  test 
players;  the  LUT  test  officer,  analyst,  data  collectors, 
and  data  authentication  group  monitored  training  and 
missions  to  learn  firsthand  the  AB3’s  capabilities. 
FDT&E  video  was  used  for  orientation  and  training 
material,  which  was  especially  beneficial  for  video  data 
reducers  who  were  not  familiar  with  recon/attack 
operations. 

Simulator  benefits 

•  Minimal  risk  to  crew  members, 

•  a  “low”  overall  test  risk  assessment, 

•  significantly  lower  test  cost, 

•  larger  data  collection  availability, 

•  reduced  test  team,  player,  and  equipment  re¬ 
quirement, 

•  less  restrictive: 

o  existing  human  use  committee  approval, 
o  aircraft  airworthiness  certification  not  re¬ 
quired. 

Conclusion 

As  evident  in  the  AB3,  simulation’s  role  in  test  and 
evaluation  is  that  of  a  key  facilitator  in  the  acquisition 
strategies  of  MBT&E  and  integrated  testing.  Though 
simulation  cannot  (and  should  not)  be  used  exclusively 
as  a  testing  application,  it  offers  significant  reductions 
to  risk  and  cost  and  increased  innovative  benefits  when 
supplementing  field  testing,  especially  when  the  system 
is  not  mature.  The  EDS  met  and  exceeded  the  test 
objective  of  TTP  refinement  by  providing  several  other 
MUM  products  and  valuable  training  material.  Critical 
pilot  workload  and  situation  awareness  differentials 
were  identified  that  were  essential  in  developing 
efficient  and  safe  employment  of  the  AB3.  Most 
notably,  the  EDS  eased  the  paradigm  shift  of 
reconnaissance  and  attack  aviation  operations  from 
autonomous  weapons  team  to  MUM  teaming. 

Test  products 

•  Level  3  database 

•  Abbreviated  operational  test  report 


•  Revised  TTP  manual 

•  MUM  platform  (handshake)  checklist 

•  Operational  test  agency  milestone  C  assessment 
report  input 

•  Recorded  mission  vignettes  (LUT  training  re¬ 
fresher) 


Residual  benefits 

•  MUM  airspace  management  and  engagement 
geometry 

•  Vetted  AB3  training  program 

•  Identification  of  tactical  suitability  and  practica¬ 
bility  of  LOIs 

•  Standardization  of  MUM  terminology  □ 
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serving  in  United  States  Army  Operational  Test  Com¬ 
mand \  Aviation  Test  Directorate,  West  Fort  Hood,  Texas. 
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Senior  Aviator  Wings,  Combat  Action  Badge,  Meritorious 
Service  Medal,  and  Air  Medal.  He  earned  a  bachelor  of 
science  degree  in  mechanical  engineering  from  Tuskegee 
University  and  a  masters  degree  in  aeronautical  science 
from  Embry-Riddle  Aeronautical  University.  E-mail: 
Cornelius,  allenjr@us.army.  mil 
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Simulation  in  T&.E 


Figure  5.  FDT&E  test  team  (photo  by  Mike  Goettings). 
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LIVE -VIRTUAL- 
CONSTRUCTIVE 
CONFERENC 

“Test  &  Evaluation 

January  24  -  27, 2011 

Wyndham  El  Paso  Airport  Hotel 
2027  Airway  Blvd.,  El  Paso,  Texas 


MARK  YOUR  CALENDAR 


TOPICS 

•  Live  Testing 

•  Virtual  Testing 

•  Testing  the  Cyber  Infrastructure 

•  System  Engineering  vs  SoS  Engineering 

•  VV&A  of  LVC  Distributed  Environments 

•  Distributed  T&E 

•  LVC  in  an  Urban  Environment 

•  Unmanned  and  Autonomous  System  Testing 

GOLF  TOURNAMENT 

The  annual  tournament  will  be  held  at  the 
Butterfield  Train  Golf  Club  on  Monday,  January  24. 
Butterfield  Trail  was  recently  voted  "Top  10  New 
Courses  You  Can  Play"  by  GOLF  Magazine.  More 
details  will  be  available  on  the  itea  website. 

EXHIBITS  and  SPONSORSHIP  Opportunities  Available 

LODGING 

The  Wyndham  El  Paso  Airport 

2027  Airway  Blvd.,  El  Paso,  Texas  79925 

915-778-4241  or  800-742-7248 

www.wyndhamelpaso.com 

Ask  for  the  ITEA  rate. 


www.itea.org 


31(3)  •  September  2010  383 


